Papers with quality metrics
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
From Brain Space to Distributional Space: The Perilous Journeys of fMRI Decoding (P19-2)
Copied to clipboard
| Challenge: | Recent work in cognitive neuroscience has introduced models for predicting distributional word meaning representations from brain imaging data. |
| Approach: | They propose to use several alternative measures to evaluate the predicted distributional space against a corpus-derived distributional spatial space. |
| Outcome: | The proposed model performs poorly on the most common metrics, while still delivering promising results. |
MENLI: Robust Evaluation Metrics from Natural Language Inference (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent proposed BERT-based evaluation metrics for text generation are vulnerable to adversarial attacks, e.g., relating to information correctness. |
| Approach: | They propose to use BERT-based evaluation metrics for text generation to evaluate text for semantic similarity but are vulnerable to adversarial attacks using Natural Language Inference. |
| Outcome: | The proposed metrics outperform existing summarization metrics but perform below SOTA MT metrics on standard benchmarks. |
Bias Mitigation in Machine Translation Quality Estimation (2022.acl-long)
Copied to clipboard
| Challenge: | despite advances in machine translation, the accuracy and fluency of translations cannot be guaranteed without a reference translation. |
| Approach: | They propose to use auxiliary tasks to mitigate partial input bias . they aim to train a multitask architecture with an auxiliary binary classification task . |
| Outcome: | The proposed models reduce partial input bias while maintaining the overall performance. |
On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Modern unsupervised machine translation systems reach reasonable translation quality under clean and controlled data conditions. |
| Approach: | They compare unsupervised and supervised machine translation systems of similar quality . they combine the benefits of both methods into a single system . |
| Outcome: | The proposed system improves adequacy and fluency as measured by human evaluators. |
Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement (2020.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on agreement between annotators, which implies a single correct interpretation. |
| Approach: | They propose an agreement-independent quality metric based on answer-coherence to evaluate on expected disagreement. |
| Outcome: | The proposed model shows that agreement is the most important indicator of quality in semantic annotation tasks. |
Diverse Keyphrase Generation with Neural Unlikelihood Training (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in neural natural language generation have made possible remarkable progress on the task of keyphrase generation, however, the importance of diversity in keyphrases has been largely ignored. |
| Approach: | They propose to train a sequence-to-sequence keyphrase generation model from the perspective of diversity. |
| Outcome: | The proposed model achieves large diversity gains while maintaining competitive output quality. |
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models (2025.acl-long)
Copied to clipboard
Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, Conghui He
| Challenge: | composition of pre-training datasets for large language models remains undisclosed . current methods for evaluating data quality are limited by single-dimensional evaluation or redundancy-focused strategies. |
| Approach: | They propose a multi-dimensional data selection method that integrates dimensions with existing quality metrics through learned optimal weightings. |
| Outcome: | The proposed method doubles convergence speed for 1.3B model models and improves downstream task performance by 3.23%. |
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization (2023.acl-long)
Copied to clipboard
Griffin Adams, Bichlien Nguyen, Jake Smith, Yingce Xia, Shufang Xie, Anna Ostropolets, Budhaditya Deb, Yuan-Jyue Chen, Tristan Naumann, Noémie Elhadad
| Challenge: | Summarization models are trained to maximize the likelihood of a single reference (MLE) but little is known about why one setup is more effective than another . |
| Approach: | They add a calibration step which exposes a model to its own ranked outputs to improve relevance or contrasts positive and negative sets to improve faithfulness. |
| Outcome: | The proposed calibration step can unlock large gains in relevance or faithfulness. |
Proxy Indicators for the Quality of Open-domain Dialogues (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of open-domain dialogues are expensive and require human annotators to evaluate their quality. |
| Approach: | They propose to use a deep-learning model trained on the general language understanding evaluation benchmark to serve as a quality indication of open-domain dialogues. |
| Outcome: | The proposed model can infer various quality metrics and derive a component-based overall score. |
Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. |
| Approach: | They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets. |
| Outcome: | The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM. |
NORMSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to understand acceptable behavior have focused on a single culture and manually built datasets from non-conversational settings. |
| Approach: | They propose a framework to automatically extract culture-specific norms from multi-lingual conversations. |
| Outcome: | The proposed framework extracts culture-specific norms from multi-lingual conversations. |
AlphaContext: An Evolutionary Tree-based Psychometric Context Generator for Creativity Assessment (2026.acl-long)
Copied to clipboard
Yixuan Wang, Yue Huang, Hong Qian, Yunzhao Wei, Yifei Ding, Wenkai Wang, Zhi Liu, Zhongjing Huang, Aimin Zhou, Jiajun Guo
| Challenge: | Existing LLM-based tools struggle with insufficient assessment cues, weak narrative coherence, limited stylistic diversity, and poor support for creative thinking. |
| Approach: | They propose an evolutionary tree-based psychometric context generator that integrates rule-guided outline planning, sentence-level MCTS generation, MAP-Elites quality-diversity optimization and assessment-guide refiner simulation. |
| Outcome: | The proposed tool outperforms strong LLMs and structured frameworks on 7 evaluation dimensions and shows higher alignment with expert-designed contexts. |
UrduMASD: A Multimodal Abstractive Summarization Dataset for Urdu (2024.lrec-main)
Copied to clipboard
| Challenge: | a surge of multimodal content on social media has transformed our methods of communication and information exchange. |
| Approach: | They propose a video-based Urdu multimodal abstractive text summarization dataset . it uses a variety of evaluation metrics to ensure the quality of the dataset amounted to a high quality one . |
| Outcome: | The proposed dataset surpasses existing datasets on key quality metrics. |